Repository navigation
Conversation
Take `OffsetSizeBytes` by value in `unpack_u32_at_offset`, remove `unpack_u32` in favor of it, and extract `str_from_bytes` from `string_from_slice`. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Add `VariantView`, `VariantMetadataView`, `VariantObjectView` and `VariantListView`. Building a view reads one header. Each accessor reads and bounds-checks only the bytes it needs, and compares field names as bytes. `VariantMetadata`, `VariantObject` and `VariantList` are now built by promoting a view, and their accessors delegate to it. `ObjectBuilder::finish` sorts fields with the new `MetadataBuilder::cmp_field_names`. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
`variant_get` reads each row of an unshredded column through `VariantView::get_path`. A row reads only the headers and field names along the path, and decodes only the value that it finds. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This was referenced Nov 18, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
This PR is stacked on #12308 and #12309. Its own change is the third commit, "perf(variant): read unshredded columns through views in
variant_get". It is a draft until they merge.Rationale for this change
For a path over unshredded data,
variant_getbuilds aVariantfor every row and then walks the path through it. Building theVariantchecks the first and last offsets of the metadata and of each object and list, and each field lookup validates the UTF-8 of the names that it probes. Rows that do not contain the path do all of that and produce a null.What changes are included in this PR?
VariantView::get_pathfollows a path of field names and list indexes through the views of feat(variant): add header-only views of encoded variant bytes #12309.variant_getreads a column that has notyped_valuethrough it, for field and index paths and for typed andVariantoutput. A row reads only the headers and names along the path, and decodes only the value that it finds.BinaryColumnresolves the array type of themetadataandvaluecolumns once, not on every row.The gain comes from skipping the per-row
Variantconstruction and the UTF-8 validation of probed names.Are these changes tested?
The existing
variant_gettests cover the new path, because their unshredded input now takes it.VariantView::get_pathhas a test that compares it withVariant::get_path.Benchmarks (local run, this PR against #12309, x86_64 Linux):
variant_get_binary_from_stringvariant_get_binary_from_unshredded_binaryvariant_get_list_index_all_oob_int64_64_rowsvariant_get_list_index_shreddedvariant_get_list_index_unshreddedvariant_get_shredded_utf8variant_get_unshredded_object_path_262k_rowsvariant_get_utf8_from_datevariant_get_utf8_from_decimalvariant_get_utf8_from_floatvariant_get_utf8_from_intvariant_get_utf8_from_listvariant_get_utf8_from_map_in_listvariant_get_utf8_from_timevariant_get_utf8_from_timestamp_with_timezonevariant_get_utf8_from_timestamp_without_timezonevariant_get_utf8_from_unshredded_stringNo benchmark in this set is slower. The shredded benchmarks (
variant_get_shredded_utf8,variant_get_list_index_shredded,variant_get_list_index_all_oob_int64_64_rows) do not take the new path.Are there any user-facing changes?
New public API in
parquet-variant:VariantView::get_path.On malformed variant bytes,
variant_geton an unshredded column returns an error, or a null under safe casting. Before, this path could panic.🤖 Generated with Claude Code